The technology described herein relates to graphics processing systems, and in particular to graphics processing systems that provide images for display for virtual reality (VR) and/or augmented reality (AR) (head mounted) display systems.
FIG. 1 shows an exemplary system on chip (SoC) graphics processing system 10 that comprises a host processor comprising a central processing unit (CPU) 1, a graphics processing unit (GPU) 2, a display controller 4, and a memory controller 6. The exemplary graphics processing system 10 may also comprise a video engine 3. As shown in FIG. 1, these units communicate via an interconnect 5 and have access to off-chip memory 7. In this system, the graphics processing unit (GPU) 2 will render frames (images) to be displayed, and the display controller 4 will then provide the frames to a display panel 8 for display.
In use of this system, an application such as a game, executing on the host processor (CPU) 1 will, for example, require the display of frames on the display 8. To do this, the application will submit appropriate commands and data to a driver for the graphics processing unit (GPU) 2 that is executing on the CPU 1. The driver will then generate appropriate commands and data to cause the graphics processing unit (GPU) 2 to render appropriate frames for display and to store those frames in appropriate frame buffers, e.g. in the main memory 7. The display controller 4 will then read those frames into a buffer for the display from where they are then read out and displayed on the display panel of the display 8.
The graphics processing system 10 will be configured to provide frames for display, and the graphics processing unit (GPU) 2 will correspondingly be configured to render frames, at an appropriate rate, such as 30 frames per second.
An example of a use of a graphics processing system such as that illustrated in FIG. 1 is to provide a virtual reality (VR) or augmented reality (AR) head mounted display (HMD) system. In this case, the display 8 will be a head-mounted display of some kind.
In a head mounted display operation, appropriate frames (images) to be displayed to each eye will be rendered by the graphics processing unit (GPU) 2 in response to appropriate commands and data from the application, such as a game, (e.g. executing on the CPU 1) that requires the display.
In such arrangements, the system will also operate to track the movement of the head/gaze of the user (so-called head pose (orientation) tracking). This head orientation (pose) data is then used to determine how the images should actually be displayed to the user for their current head position (view orientation (pose)), and the images (frames) are rendered accordingly (for example by setting the camera orientation (viewpoint and view direction) based on the head orientation data), so that an appropriate image (frame) based on the user's current direction of view can be displayed.
While it would be possible simply to determine the head orientation (pose) at the start of the graphics processing unit (GPU) 2 rendering a frame to be displayed in a virtual reality (VR) or augmented reality (AR) system, and then to update the display 8 with the frame once it has been rendered, because of latencies in the rendering process, it can be the case that the user's head orientation (pose) has changed between the sensing of the head orientation (pose) at the beginning of the rendering of the frame and the time when the frame is actually displayed (scanned out to the display 8). Moreover, it is often desirable to be able to provide frames for display in a virtual reality (VR) or augmented reality (AR) system at a rate that is faster than the graphics processing unit (GPU) 2 may be able to render frames at.
To allow for this, a process known as “timewarp” has been proposed for head mounted display systems. In this process, an “application” frame is first rendered by the graphics processing unit (GPU) 2 based on the head orientation (pose) data sensed at the beginning of the graphics processing unit (GPU) 2 rendering the application frame, but then before an image is actually displayed on the display 8, further head orientation (pose) data is sensed, and that updated head orientation (pose) sensor data is used transform the graphics processing unit (GPU) 2 rendered application frame to generate an “updated” version of the application frame that takes account of the updated head orientation (pose) data. The so-“timewarped” updated version of the application frame is then displayed on the display 8.
The processing required to “timewarp” a graphics processing unit (GPU) 2 rendered application frame can typically be performed in a much shorter time than the time required for the graphics processing unit (GPU) 2 to render a frame. Thus by performing “timewarp” processing, the time between head orientation (pose) data being sensed, and the image displayed on the display 8 being updated using the sensed head orientation (pose) data, can be reduced as compared to the graphics processing unit (GPU) 2 directly rendering each image to be displayed on the display 8 without “timewarp” processing. The effect of this is that, by using “timewarp” processing, the image displayed on the display 8 can more closely match the user's latest head orientation (pose), resulting in a more realistic virtual reality (VR) or augmented reality (AR) experience, for example.
Similarly, “timewarp” processing can be performed at a faster rate, such as 90 or 120 frames per second, than the graphics processing unit (GPU) 2 may be able to render frames at, such as 30 frames per second. Thus, “timewarp” processing can be used to provide frames for display that have been updated based on a sensed head orientation (pose) at a faster rate than would otherwise be possible without the use of “timewarp” processing. This can help to reduce “judder” artefacts and provide a smoother virtual reality (VR) or augmented reality (AR) experience, for example.
FIGS. 2, 3 and 4 illustrate the “timewarp” process in more detail.
FIG. 2 shows the display of an exemplary frame 20 when the viewer is looking straight ahead, and the required “timewarp” projection of that frame 21 when the viewing angle of the user changes due to a head rotation. It can be seen from FIG. 2 that for the frame 21, a modified version of the frame 20 must be displayed.
FIG. 3 correspondingly shows the “timewarp” rendering 31 of application frames 30 to provide the “timewarped” frames 32 for display. As shown in FIG. 3, a given application frame 30 that has been rendered may be subject to two (or more) “timewarp” processes 31 for the purpose of displaying the appropriate “timewarped” version 32 of that application frame 30 at successive intervals whilst waiting for a new application frame to be rendered. The “timewarp” processing 31 can be performed in parallel with (using a different thread to) the rendering of application frames 30 (i.e. asynchronously), which is referred to as “asynchronous timewarp” (ATW) processing.
FIG. 3 also shows the regular sampling 33 of the head orientation (pose) data that is used to determine the appropriate “timewarp” modification that should be applied to an application frame 30 for displaying the frame appropriately to the user based on their head orientation (pose).
FIG. 4 shows a schematic illustration of an exemplary rendered application frame 40, together with four respective “timewarped” frames 41A-D generated for display by “timewarping” application frame 40. So as to ensure that graphics processing unit (GPU) 2 rendered application frame data is available to be “timewarped” for a range of possible head movements, application frames are typically rendered by the graphics processing unit (GPU) 2 based on a field of view that is wider than the field of view of the output “timewarped” frames that are actually displayed to the user. The field of view of an application frame may be based on, for example, a permitted or expected (maximum) amount of head motion (e.g. rotation and/or translation) in the time period that the application frame is supposed to be valid for. Then, when the application frame is to be displayed, the “timewarp” process will be used to effectively render an appropriate window (“letterbox”) taken from the wider field of view of the application frame. Thus in the schematic example of FIG. 4, application frame 40 is provided by rendering the scene based on a 16×8 square field of view, but “timewarped” frames 41A-D are provided for display with only a 5×4 square field of view taken from the wider field of view.
Each “timewarped” frame will also be transformed (“timewarped”), e.g. as described above, based on more recent head orientation (pose) information to provide the actual output image that is displayed to the use. Thus, as shown in the example of FIG. 4, when a change in head orientation (pose) is detected, application frame 40 is transformed such that object 42 appears at an appropriately shifted position in “timewarped” frames 41B-D, as compared to when no change in head orientation (pose) is detected (41A). Thus, as shown in FIG. 4, when a head movement to the right is detected, object 42 appears shifted to the left (41B); when a larger head movement to the right is detected, object 42 appears shifted farther to the left (41C); and when a head movement to the left is detected, object 42 appears shifted to the right (41D) as compared to when no change in head orientation (pose) is detected (41A).
Thus, in “timewarp” processing, an application frame is first rendered based on a first view orientation (pose) sensed at the beginning of rendering the application frame, and thus essentially represents a static “snapshot” of the scene being rendered as it should appear to a user at the point in time that the first view orientation (pose) was sensed. “Timewarp” processing can then be used to update (transform) the static “snapshot” application frame based on one or more second view orientations (poses) sensed at one or more respective later points in time, after the application frame has been rendered, to provide a series of one or more successive “timewarped” frames that each represent an updated view of the scene at the respective later point in time.
It has been recognised that while such “timewarp” processing takes account of changes to view orientation (pose) during the time period between the point in time at which the first view head orientation (pose) is sensed, and the point in time at which a respective second view orientation (pose) is sensed, it does not account for, and so “timewarped” frames do not show, any changes due to the motion of objects within the scene during that same time period. This means that the “timewarp” processing of a rendered application frame that represents a dynamic scene, i.e. a scene that includes moving objects, can introduce distortions in what is displayed to a user.
To account for object motion when performing “timewarp” processing, a process known as “spacewarp” processing has been proposed. This process attempts to take account of any motion of objects when a “timewarped” frame is to be generated by “timewarping” an application frame based on a view orientation (pose) sensed at a later point in time, by extrapolating moving objects shown in the application frame to expected e.g. positions at that later point in time, with the “timewarp” processing then being performed on the basis of the extrapolated objects. The so-“timewarped” and “spacewarped” updated version of the application frame is then displayed on the display 8.
The Applicants believe that there remains scope for improvements to graphics processing systems, and in particular to graphics processing systems that provide “timewarped” and/or “spacewarped” images for display for virtual reality (VR) and/or augmented reality (AR) (head mounted) display systems.
Like reference numerals are used for like components where appropriate in the drawings.